Methods and Systems for Dynamic Sampling of Application Events

ABSTRACT

A method of dynamically sampling application events includes receiving usage data for a plurality of applications on client devices. The usage data includes data for instances of events that occur in the plurality of applications in response to users interacting with the plurality of applications on the client devices. The method further includes storing the usage data in a first data store, determining, for respective applications of the plurality of applications, sampling rates for respective application events of the plurality of application events based at least in part on respective quantities of instances of the respective application events during a period of time, sampling from the first data store the data for the instances of the respective application events at the sampling rates to form respective sets of usage data, and providing a report using one or more sets of the sets of usage data.

RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.15/432,753, filed Feb. 14, 2017, entitled “Methods and Systems forDynamic Sampling of Application Events,” which is incorporated byreference herein in its entirety.

TECHNICAL FIELD

This relates generally to data sampling, including but not limited todynamically sampling of application events.

BACKGROUND

Software applications generate large volumes of data providing valuableinsights into user behavior within the software applications themselves.However, the sheer volume of data generated by software applications canbe overwhelming, even to data-analyzing computer programs. Consequently,interested parties (e.g., online service providers, advertisers, etc.)may benefit from processes that effectively use computer capabilities toanalyze large volumes of data generated by users interacting withsoftware applications.

SUMMARY

Accordingly, there is a need for methods and systems that caneffectively analyze (e.g., organize and extract useful statistics from)large volumes of data generated by users interacting with softwareapplications.

In accordance with some embodiments, a method is performed at a serversystem having one or more processors and memory storing instructions forexecution by the processors. The method includes receiving usage datafor a plurality of applications on client devices. The usage dataincludes data for instances of each of a plurality of application eventsthat occur in the plurality of applications in response to usersinteracting with the plurality of applications on the client devices.The method further includes storing the usage data in a first datastore; determining, for respective applications of the plurality ofapplications, sampling rates for respective application events of theplurality of application events based at least in part on respectivequantities of instances of the respective application events during aperiod of time; and sampling from the first data store the data for theinstances of the respective application events at the sampling rates toform respective sets of usage data. Each set of usage data includes datasamples corresponding to distinct instances of a respective applicationevent. The method also includes storing the sets of usage data in asecond data store and providing a report using one or more sets of thesets of usage data.

In accordance with some embodiments, a server system includes one ormore processors/cores and memory storing one or more programs configuredto be executed by the one or more processors/cores. The one or moreprograms include instructions for performing the operations of themethod described above. In accordance with some embodiments, anon-transitory computer-readable storage medium has stored thereininstructions that when executed by one or more processors/cores of aserver system, cause the server system to perform the operations of themethod described above.

These dynamic sampling techniques provide a server system with processesto effectively retrieve statistics from the large volumes of data.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures and specification.

FIG. 1 is a block diagram illustrating an exemplary networkarchitecture, in accordance with some embodiments.

FIG. 2 is a block diagram illustrating an exemplary server system, inaccordance with some embodiments.

FIGS. 3A-3F are flow diagrams illustrating a method of dynamic samplingof application events, in accordance with some embodiments.

FIG. 4 is an illustration of a report provided by the server system, inaccordance with some embodiments.

FIG. 5 is an illustration of a report provided by the server system, inaccordance with some embodiments.

DESCRIPTION OF EMBODIMENTS

Reference will now be made to embodiments, examples of which areillustrated in the accompanying drawings. In the following description,numerous specific details are set forth in order to provide anunderstanding of the various described embodiments. However, it will beapparent to one of ordinary skill in the art that the various describedembodiments may be practiced without these specific details. In otherinstances, well-known methods, procedures, components, circuits, andnetworks have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are used onlyto distinguish one element from another. For example, a first samplingrate could be termed a second sampling rate, and, similarly, a secondsampling rate could be termed a first sampling rate, without departingfrom the scope of the various described embodiments. The first samplingrate and the second sampling rate are both sampling rates, but they arenot the same sampling rate unless specified otherwise.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if′ is, optionally, construed to mean “when”or upon” or “in response to determining” or “in response to detecting”or “in accordance with a determination that,” depending on the context.Similarly, the phrase “if it is determined” or “if [a stated conditionor event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event]” or “in accordance with a determination that [astated condition or event] is detected,” depending on the context.

As used herein, the term “exemplary” is used in the sense of “serving asan example, instance, or illustration” and not in the sense of“representing the best of its kind.”

FIG. 1 is a block diagram illustrating an exemplary network architecturein accordance with some embodiments. The network architecture 100includes a number of client devices 104-1, 104-2, . . . 104-ncommunicably connected to a social networking system 108 by one or morenetworks 106.

In some embodiments, the client devices 104-1, 104-2, . . . 104-n arecomputing devices such as smart watches, personal digital assistants,portable media players, smart phones, tablet computers, 2D gamingdevices, 3D gaming devices, virtual reality devices, laptop computers,desktop computers, televisions with one or more processors embeddedtherein or coupled thereto, in-vehicle information systems (e.g., anin-car computer system that provides navigation, entertainment, and/orother information), or other appropriate computing devices that can beused to communicate with an electronic social networking system 108 andother computing devices (e.g., via the social networking system). Insome embodiments, the social networking system 108 is a single computingdevice such as a computer server, while in other embodiments, the socialnetworking system 108 is implemented by multiple computing devicesworking together to perform the actions of a server system (e.g., cloudcomputing). In some embodiments, the network 106 is a publiccommunication network (e.g., the Internet or a cellular data network), aprivate communications network (e.g., private LAN or leased lines), or acombination of such communication networks.

Users 102-1, 102-2, . . . 102-n employ the client devices 104-1, 104-2,. . . 104-n to execute software applications (e.g., “apps”) that arespecific to third-party servers 110 and/or the social networking system108. When employing the client devices 104 to execute softwareapplications, users 102 generate usage data (e.g., application events)by interacting with features (e.g., selecting affordances, swiping inparticular directions, etc.) displayed in the software applications. Thevarious software applications may involve numerous distinct applicationevents. For example, in a gaming application, a user may generate afirst application event by completing a level within a game and maygenerate a second application event by obtaining a threshold scorewithin the game. In another example, in a retail application, a user maygenerate a first application event by adding an item to his or hervirtual shopping cart and may generate a second application event bypaying or failing to complete payment for the item in the virtualshopping cart. In another example, in a sports application, a user maygenerate an application event by adding a player to his or her fantasysports team. In addition, users may generate application events byinstalling an application, uninstalling an application, launching anapplication, and the like.

In some embodiments, the network architecture 100 also includes one ormore third-party servers 110. In some embodiments, third-party servers110 are associated with third-party service providers who provideservices and/or features to users of a network (e.g., users of thesocial networking system 108, FIG. 1 ). In some embodiments, a giventhird-party server 110 is used to host third-party software applicationsthat are used by client devices 104, either directly with, inconjunction with, or independently of the social networking system 108(e.g., the gaming, retail, and/or sports applications discussed above).For example, an application executing on a client device 104 maycommunicate information, via the client device 104, regarding activity(e.g., application events) in the application to the third-party server110. The third-party server 110 may in turn communicate the informationto the social networking system 108. Alternatively, in some embodiments,the client device 104 communicates the information directly to thesocial networking system 108. One skilled in the art will appreciatethat various techniques may be used to gather information regardingactivity in an application executing on a client device.

FIG. 2 is a block diagram illustrating an exemplary server system 200 inaccordance with some embodiments. In some embodiments, the server system200 is an example of the social networking system 108 (FIG. 1 ). Theserver system 200 typically includes one or more processing units(processors or cores) 202, one or more network or other communicationsinterfaces 204, memory 206, and one or more communication buses 208 forinterconnecting these components. The communication buses 208 optionallyinclude circuitry (sometimes called a chipset) that interconnects andcontrols communications between system components. The server system 200optionally includes a user interface (not shown). The user interface, ifprovided, may include a display device and optionally includes inputssuch as a keyboard, mouse, trackpad, and/or input buttons. Alternativelyor in addition, the display device includes a touch-sensitive surface,in which case the display is a touch-sensitive display.

Memory 206 includes high-speed random access memory, such as DRAM, SRAM,DDR RAM, or other random access solid state memory devices; and mayinclude non-volatile memory, such as one or more magnetic disk storagedevices, optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 206 may optionallyinclude one or more storage devices remotely located from theprocessor(s) 202. Memory 206, or alternately the non-volatile memorydevice(s) within memory 206, includes a non-transitory computer readablestorage medium. In some embodiments, memory 206 or the computer readablestorage medium of memory 206 stores the following programs, modules, anddata structures, or a subset or superset thereof:

-   -   an operating system 210 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 212 that is used for connecting        server system 200 to other computers (e.g., client devices        104-1, 104-2, . . . 104-n, and/or third party server(s) 110) via        the one or more communication network interfaces 204 (wired or        wireless) and one or more communication networks, such as the        Internet, cellular telephone networks, mobile data networks,        other wide area networks, local area networks, metropolitan area        networks, and so on;    -   a server database 214 for storing data associated with the        server system 200, such as:        -   a first data store 216;        -   a second data store 218;        -   dashboards 220; and        -   one or more lists 222.    -   a sample module 228 for sampling application events stored in        the first and/or second data stores 216 and 218 using one or        more sampling rates; and    -   a report module 230 for providing a report associated with usage        data.

The server database 214 stores data (e.g., usage data received from aplurality of applications) in one or more types of databases, such astext, graph, dimensional, flat, hierarchical, network, object-oriented,relational, and/or XML databases.

The first data store 216 stores usage data for a plurality ofapplications installed on client devices. In some embodiments, the usagedata is received from the client devices (e.g., client devices 104-1through 104-n, FIG. 1 ). Alternatively or in addition, in someembodiments, the usage data is received from one or more third-partyservers (e.g., third-party server(s) 110, FIG. 1 ) that collect theusage data from client devices. In some embodiments, the usage dataincludes application events that occur in the plurality of applicationsin response to users interacting with the plurality of applications onthe client devices (e.g., user 102-1 interacting with (e.g., touching,clicking, swiping, playing, etc.) an application executing on the clientdevice 104-1, FIG. 1 ).

The second data store 218 stores sets of usage data sampled from theusage data stored in the first data store 216. Each set of usage dataincludes data samples corresponding to distinct instances of arespective application event (e.g., a first set of usage data includesinstances of Event A in Application A, a second set of usage dataincludes instances of Event B in Application A, a third set of usagedata includes instances of Event A in Application B, and so on).

The usage data stored in the first data store 216 are sampled atspecified sampling rates to form the sets of usage data. In someembodiments, each application event has a distinct sampling rate (e.g.,Event A in Application A has a first sampling rate, Event B inApplication A has a second sampling rate, and so on). In someembodiments, sampling rates (e.g., the first sampling rate) may be basedat least in part on respective quantities of instances of respectiveapplication events during a period of time (e.g., a plurality of dayssuch as a week, two weeks, etc.).

In some embodiments, thresholds are used to limit (e.g., cap) an amountof samples (e.g., quantity of instances) gathered for a respective setof usage data. In this way, limits are placed on volumes of usage datastored in the second data store 218 for the respective sets of usagedata.

In some embodiments, the one or more lists 222 include a whitelist ofevents. The server system 200 may store usage data for each instance ofan application event on the whitelist in the second data store 218. Forexample, application installs or application uninstalls may beapplication events on the whitelist of events. In some embodiments, theone or more lists 222 include identified users. For example, the serversystem 200 may identify a user associated with an instance of anapplication event for an application and log the identified user in oneof the one or more lists 222. Identifying users is discussed in furtherdetail below with reference to method 300.

The dashboards 220 are interfaces that organize and present informationin a way that is easy to read and understand. The dashboards 220 includemultiple interfaces that may be used to present information. One exampleof a dashboard is illustrated by report 400 (FIG. 4 ), which showsnumbers of instances of specific events 402, 404, 406 as percentages ofall events for an application. Another example of a dashboard isillustrated by report 500 (FIG. 5 ).

In some embodiments, the sample module 228 includes a rate-settingmodule 230. The rate-setting module 230 is used for adjusting thesampling rates based on changes in received usage data (e.g., changes inrates of receiving usage data).

FIGS. 3A-3F are flow diagrams illustrating a method 300 for dynamicsampling of application events, in accordance with some embodiments. Thesteps of the method 300 (including methods 338, 344, 358, 366, and/or378, FIGS. 3C-3F) may be performed by a server system (e.g., socialnetworking system 108, FIG. 1 ; server system 200, FIG. 2 ). FIGS. 3A-3Fcorrespond to instructions stored in a computer memory orcomputer-readable storage medium (e.g., memory 206 of the server system200, FIG. 2 ). For example, the operations of the method 300 areperformed, at least in part, by the sample module 228 (FIG. 2 ) and thereport module 232 (FIG. 2 ). To assist with describing the method 300,the method 300 will be described with reference to FIG. 4 and FIG. 5 .

In performing the method 300, the server system receives (302) usagedata for a plurality of applications on client devices (e.g., clientdevices 104-1, 104-2, . . . 104-n, FIG. 1 ). The usage data includesdata for instances of each of a plurality of application events thatoccur in the plurality of applications in response to users interactingwith the plurality of applications on the client devices. For example, auser interacting with Application A may launch (e.g., Event A)Application A x-number of times (e.g., x-number of instances) and mayselect an affordance (e.g., Event B) displayed in Application A y-numberof times (e.g., y-number of instances). In response to the userinteractions, Application A may provide the server system with usagedata for the x-number of instances and the y-number of instances. Insome embodiments, the usage data further includes applicationidentification, client type (e.g., iOS, Android, etc.), applicationversion, event timestamp, and the like. In some embodiments, athird-party server (e.g., third party server 110, FIG. 10 ) receives theusage data from the client device and provides the usage data to theserver system.

The server system stores (304) the usage data in a first data store(e.g., first data store 216, FIG. 2 ). For example, the server systemstores data for the x-number of instances of Event A and the y-number ofinstances of Event B for Application A in the first data store 216.

The server system determines (306), for respective applications of theplurality of applications, sampling rates for respective applicationevents of the plurality of application events based at least in part onrespective quantities of instances of the respective application eventsduring a first period of time (e.g., a plurality of days). For example,a quantity of instances of Event B of Application A during the firstperiod of time is the y-number of instances. As such, the server systemmay base the sampling rate for Event B at least in part on the y-numberof instances. The server system may determine unique sampling rates foreach application event occurring in each respective application of theplurality of applications. For example, referring to FIG. 4 , the serversystem determines, for Application A, a first sampling rate for Event A402 (e.g., application launch), a second sampling rate for Event B 404(e.g., selection of affordance), a third sampling rate for Event C 406,and so on. The server system may determine, for other applications(e.g., Application B, Application C, and so on), a first sample rate forEvent A of each application, a second sampling rate for Event B of eachapplication, and so on.

In some embodiments, when determining the sampling rate for a respectiveapplication event, the server system determines (308) daily counts (orcounts during another specified time period) of instances of therespective application event for which data are received. For example,the server system determines daily counts of instances for each day of aplurality of days (e.g., a week). Furthermore, the server systemcalculates (310) a statistic based on the daily counts. For example, thestatistic is a median value of daily counts (312). Alternatively, insome embodiments, the statistic is a mode of daily counts, a mean ofdaily counts, a specified number of standard deviations from the mean ofdaily counts, a specified percentile of daily counts, or the like. Theserver system uses (314) the statistic to select the sampling rate. Forexample, referring to Event B and the y-number of instances, the serversystem uses the median value of daily counts of the y-number ofinstances as the statistic when determining the sampling rate for EventB.

The server system samples (316), from the first data store, the data forthe instances of the respective application events at the sampling rates(also referred to herein as data-store sampling rates) to formrespective sets of usage data. Each set of usage data includes datasamples corresponding to distinct instances of a respective applicationevent. For example, the server system may sample from usage data for thex-number of instances associated with Event A of Application A at thefirst sampling rate to form a first set of usage data and may samplefrom usage data for the y-number of instances associated with Event B ofApplication A at the second sampling rate to form a second set of usagedata. In some embodiments, the first and second sampling rates are thesame rate (e.g., coincidentally). In some embodiments, the first andsecond sampling rates are different rates.

The server system stores (318) the sets of usage data in a second datastore (e.g., second data store 218, FIG. 2 ).

The server system provides (320, FIG. 3B) a report (e.g., report 400,FIG. 4 ; report 500, FIG. 5 ) using one or more sets of the sets ofusage data. For example, referring to FIG. 5 , report 500 includes anevent column 502, and sample period of time column 504 (e.g., the firstperiod of time), and a description column 506 which includes adescription (e.g., facts, statistics, etc.) of user activity inApplication A during the sample period of time (e.g., the first periodof time). As shown for Event B, the report (e.g., description column506) includes one or more relevant facts such as a demographic statistic(e.g., 25% of users invoking Event B during the sample period of timewere female), a peak usage statistic (e.g., users invoking Event B peakson Jan. 19, 2016 from 4:00 pm to 6:30 pm), and an achievement statistic(e.g., 85% of users of Application A did not proceed to Event C afterinvoking Event B). The description column 506 of the report 500 ismerely illustrative and one skilled in the art will appreciate that anynumber of descriptors could be included in the report 500 in any format,depending on the application (e.g., retail application may include afirst set of descriptors in the description column 506 and a sportsapplications may include a second set of descriptors in the descriptioncolumn 506).

In some embodiments, to provide (320) the report, the server systemsamples (322) a first set of the one or more sets of usage data from thesecond data store at a query sampling rate to form a subset of the firstset. In this way, the server system may sample usage data from the firstdata store at the data-store sampling rates and may further sample thesets of usage data from the second data store at the query samplingrate. For example, referring to Event B of Application A, the serversystem may sample usage data associated with Event B from the first datastore at a data-store sampling rate for Event B to form a set of usagedata associated with Event B. The server system may further sample theset of usage data associated with Event B at a query sampling rate toform the subset. In addition, the server system may generate (324) thereport (e.g., report 500, FIG. 5 ) using the subset (e.g., a firstsubset).

In some embodiments, the server system further samples a second set ofthe one or more sets from the second data store at another querysampling rate to form a second subset of the second set (e.g., Event Cof Application A). The server system may generate the report using thesecond subset either alone or in combination with the first subset. Insome embodiments, the server system may generate the report using one ormore subsets (or sets) of usage data from each of a plurality ofapplications.

In some embodiments, a product of the query sampling rate and thedata-store sampling rate for an application event that corresponds to afirst set of usage data equals a predefined overall sampling rate. Forexample, the server system detects a change in the rate at which datafor instances of the first application event are received (e.g., usersof Application A may invoke Event B at a reduced rate). In response todetecting the change, the server system may change a first data-storesampling rate inversely with the change in the rate at which data forinstances of the first application event are received. For example, inif users invoke an application event at a reduced (or increased) rate,the server system may increase (or decrease) the first data-storesampling rate accordingly. Furthermore, the server system changes thequery sampling rate in accordance with the change to the firstdata-store sampling rate, to maintain the overall sampling rate. Forexample, the server system may increase (or decrease) the firstdata-store sampling rate and proportionally decrease (or increase) thequery sampling rate so that the product of the query sampling rate andthe data-store sampling rate for the application event (e.g., Event B ofApplication A) continues to equal the predefined overall sampling rate.

In some embodiments, to provide (320) the report using the one or moresets of the sets of usage data (and/or the one or more subsets), theserver system generates (326) a dashboard showing informationcharacterizing the one or more sets (and/or the one or more subsets).The reports 400 (FIG. 4 ) and 500 (FIG. 5 ) are examples of a dashboard.

In some embodiments, the server system maintains a whitelist ofapplication events (e.g., maintained in one of the lists 222, FIG. 2 )and stores, in the second data store, all received data for instances ofthe application events on the whitelist. In some embodiments,application launches, installs, and/or uninstalls are included in thewhitelist. For example, Event A is associated with launching ApplicationA and is included on the whitelist. In some embodiments, the serversystem designates application events to be included in the whitelist.Alternatively or in addition, a third party designates applicationevents to be included in the whitelist. Accordingly, to provide (320)the report using one or more sets of the sets of usage data, the serversystem generates (328) the report using the data in the second datastore for all instances of an application event on the whitelist.

In some embodiments, the server system determines (330), for therespective applications of the plurality of applications, updatedsampling rates for the respective application events of the plurality ofapplication events based at least in part on respective quantities ofinstances of the respective application events during a second period oftime that overlaps with the first period of time. For example, the firstperiod of time may be a 7-day period and the second period of time maybe another 7-day period that includes a number days included in thefirst period of time and a number of days not included in the firstperiod of time. When determining the updated sampling rates, the serversystem may use some usage data received during the first period of timeand some usage data received during the second period of time. In thisway, the server system may determine trends in usage data between andwithin periods of time.

Furthermore, in some embodiments, the server system samples (332) usagedata stored in the first data store during the second period of time atthe updated sampling rates to update the respective sets of usage dataand stores (334) the updated sets of usage data in the second datastore. The server system may provide (336) another report using one ormore sets of the updated sets of usage data. The sampling, storing, andproviding steps 332, 334, and 336 are discussed in further detail abovewith reference to steps 316, 318, 320, respectively.

In some embodiments, the method 300 may include method 338 (FIG. 3C).The steps of the method 338 may be performed in conjunction with thesteps of the method 300. For example, the steps of the method 338 may beperformed between steps 306 and 316 of the method 300.

In performing the method 338, the server system detects (340) a changein the rate at which data (i.e., usage data) for instances of arespective application event are received. For example, users ofApplication A may invoke Event B at a lesser rate as a period of timeprogresses. Accordingly, the server system may detect the change in therate at which data for instances of Event B are received.

In response to detecting the change in the rate, the server systemchanges (342) the sampling rate for the respective application eventinversely with the change in the rate at which data for instances of therespective application event are received. For example, if therespective application event is invoked at a lesser (or greater) rate asthe period of time progresses (e.g., data for fewer instances of Event Bof Application A are received during a later portion of the period oftime relative to data for an amount of instances of Event B receivedduring an earlier portion of the period of time), the server systemincreases (or decreases) the sampling rate of the respective applicationevent.

In some embodiments, the method 300 may include method 344 (FIG. 3D).The steps of the method 344 may be performed in conjunction with thesteps of the method 300. For example, the steps of the method 344 may beperformed during step 316 of the method 300.

In performing the method 344, the server system determines (346) whethera quantity of data samples in a respective set of usage data in thesecond data store satisfies (e.g., is greater than, or greater than orequal to) a first threshold. In some embodiments, the first threshold isspecific to the respective set of usage data (e.g., specific to the setof usage data for Event B of Application A). In some embodiments, thefirst threshold is a buffer threshold. For example, the first thresholdmay be set to a value below a maximum threshold (e.g., set 20% below themaximum threshold).

If the quantity of data samples in the respective set of usage data inthe second data store does not satisfy the first threshold (346-No), theserver system continues (348) sampling at the sampling rate for theapplication event that corresponds to the respective set of usage data(e.g., continues sampling data for instances of Event B of Application Aat the sampling rate for the set of usage data associated with Event B).

If the quantity of data samples in the respective set of usage data inthe second data store satisfies the first threshold (346-Yes), theserver system reduces (350) the sampling rate for the application eventthat corresponds to the respective set of usage data (e.g., continuessampling data for instances of Event B of Application A but does so at areduced sampling rate for the set of usage data associated with EventB).

The server system may determine (352) whether the quantity of datasamples in the respective set of usage data in the second data storesatisfies (e.g., is greater than, or greater than or equal to) a secondthreshold greater than the first threshold. In some embodiments, thesecond threshold is a maximum threshold.

If the quantity of data samples in the respective set of usage data inthe second data store does not satisfy the second threshold (352-No),the server system continues (354) sampling at the reduced sampling rate(e.g., continues sampling instances of Event B of Application A at thereduced sampling rate for the set of usage data associated with EventB).

If the quantity of data samples in the respective set of usage data inthe second data store satisfies the second threshold (352-Yes), theserver system ceases (356) sampling, from the first data store, the datafor instances of the application event that corresponds to therespective set of usage data (e.g., stops sampling instances of Event Bof Application A for the set of usage data associated with Event B).

In some embodiments, the method 300 may include method 358 (FIG. 3E).The steps of the method 358 may be performed in conjunction with thesteps of the method 300. For example, the steps of the method 358 may beperformed during step 316 of the method 300.

In performing the method 358, the server system determines (360) whethera quantity of data samples in a respective set of usage data in thesecond data store satisfies (e.g., is greater than, or greater than orequal to) a threshold. In some embodiments, the threshold is a bufferthreshold (e.g., set below a maximum threshold). In some embodiments,the threshold is a maximum threshold. If the quantity of data samples inthe respective set of usage data in the second data store does notsatisfy the threshold (360-No), the server system continues (362)sampling at the sampling rate for the application event that correspondsto the respective set of usage data (e.g., continues sampling instancesof Event B of Application A at the sampling rate for the set of usagedata associated with Event B).

If the quantity of data samples in the respective set of usage data inthe second data store satisfies the threshold (360-Yes), the serversystem ceases (364) sampling, from the first data store, the data forinstances of the application event that corresponds to the respectiveset of usage data (e.g., stops sampling instances of Event B ofApplication A for the set of usage data associated with Event B).

In some embodiments, the method 300 may include method 366 (FIG. 3F).The steps of the method 366 may be performed in conjunction with thesteps of the method 300. For example, the steps of the method 366 may beperformed after or during steps 302 and/or 304 of the method 300.

In performing the method 366, the server system identifies (368) a userassociated with an instance of an application event for an applicationand stores data for the instance in the second data store as part of thesampling step 316 and storing step 318. In some embodiments, the serversystem, when identifying the user, selects the user at random afterreceiving an instance of an application event (e.g., Event B) associatedwith an application (e.g., Application A).

In some embodiments, to identify (368) the user, the server systemdetermines (370) an account identifier of the user. For example, theaccount identifier of the user may be an account associated with asocial networking service provided by the server system. Alternatively,the account identifier of the user may be associated with the respectiveapplication event (e.g., Event B may be an event for an application inwhich the user has an identifiable account).

In some embodiments, when identifying the user, the server systemdetermines (372) an address of a client device of the user. For example,the address of the client device of the user may be an Internet Protocol(IP) address (374).

The server system stores (376), in the second data store, all subsequentreceived data for instances of application events associated with theuser for the application (or for multiple applications, such as allmonitored applications). Accordingly, the server system may track usagedata for particular users (e.g., for a single application or acrossmultiple applications).

Although some of various drawings illustrate a number of logical stagesin a particular order, stages which are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beobvious to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the claims to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The embodiments were chosen in order to best explain theprinciples underlying the claims and their practical applications, tothereby enable others skilled in the art to best use the embodimentswith various modifications as are suited to the particular usescontemplated.

What is claimed is:
 1. A method, comprising: at a server system havingone or more processors and memory storing instructions for execution bythe one or more processors: receiving usage data for a plurality ofapplications on client devices, wherein the usage data includes data forinstances of each of a plurality of application events that occur in theplurality of applications, in response to users interacting with theplurality of applications on the client devices; storing the usage datain a first data store; determining, for respective applications of theplurality of applications, sampling rates for respective applicationevents of the plurality of application events based at least in part onrespective quantities of instances of the respective application eventsduring a period of time; after determining the sampling rates: sampling,from the first data store, the data for the instances of the respectiveapplication events at the sampling rates to form respective sets ofusage data, wherein each set of usage data includes data samplescorresponding to distinct instances of a respective application event;storing the sets of usage data in a second data store; and providing areport using one or more sets of the sets of usage data.
 2. The methodof claim 1, wherein the period of time is a first period of time and themethod further comprises, at the server system: determining, for therespective applications of the plurality of applications, updatedsampling rates for the respective application events of the plurality ofapplication events based at least in part on respective quantities ofinstances of the respective application events during a second period oftime that overlaps with the first period of time; after determining theupdated sampling rates: sampling usage data stored in the first datastore during the second period of time at the updated sampling rates toupdate the respective sets of usage data; storing the updated sets ofusage data in the second data store; and providing another report usingone or more sets of the updated sets of usage data.
 3. The method ofclaim 1, wherein the sampling rates are data-store sampling rates andproviding the report comprises: sampling a first set of the one or moresets from the second data store at a query sampling rate to form asubset of the first set; and generating the report using the subset. 4.The method of claim 3, wherein a product of the query sampling rate andthe data-store sampling rate for the application event that correspondsto the first set equals a predefined overall sampling rate.
 5. Themethod of claim 4, wherein the application event that corresponds to thefirst set is a first application event and the data-store sampling ratefor the first application event is a first data-store sampling rate, themethod further comprising, at the server system: detecting a change inthe rate at which data for instances of the first application event arereceived; changing the first data-store sampling rate inversely with thechange in the rate at which data for instances of the first applicationevent are received; and changing the query sampling rate in accordancewith the change to the first data-store sampling rate, to maintain theoverall sampling rate.
 6. The method of claim 1, further comprising, atthe server system: detecting a change in the rate at which data forinstances of a respective application event are received; and changingthe sampling rate for the respective application event inversely withthe change in the rate at which data for instances of the respectiveapplication event are received.
 7. The method of claim 1, furthercomprising, at the server system: maintaining a whitelist of applicationevents; and storing, in the second data store, all received data forinstances of the application events on the whitelist.
 8. The method ofclaim 7, wherein providing the report comprises generating the reportusing the data in the second data store for all instances of anapplication event on the whitelist.
 9. The method of claim 1, furthercomprising, at the server system: identifying a user associated with aninstance of an application event for an application, wherein data forthe instance is sampled and stored in the second data store; and inresponse to identifying the user, storing in the second data store allsubsequent received data for instances of application events associatedwith the user for the application.
 10. The method of claim 9, whereinidentifying the user comprises determining an account identifier for theuser.
 11. The method of claim 9, wherein identifying the user comprisesdetermining an address of a client device of the user.
 12. The method ofclaim 11, wherein the address is an Internet Protocol (IP) address. 13.The method of claim 1, further comprising, at the server system: makinga first determination that a quantity of data samples in a respectiveset of usage data in the second data store satisfies a first threshold;and in accordance with the first determination, reducing the samplingrate for the application event that corresponds to the respective set ofusage data.
 14. The method of claim 13, further comprising, at theserver system: making a second determination that the quantity of datasamples in the respective set of usage data in the second data storesatisfies a second threshold, wherein the second threshold is greaterthan the first threshold; and in accordance with the seconddetermination, ceasing to sample, from the first data store, the datafor instances of the application event that corresponds to therespective set of usage data.
 15. The method of claim 1, furthercomprising, at the server system: determining that a quantity of datasamples in a respective set of usage data in the second data storesatisfies a threshold; and in accordance with the determination, ceasingto sample, from the first data store, the data for instances of theapplication event that corresponds to the respective set of usage data.16. The method of claim 1, wherein the period of time includes aplurality of days and determining the sampling rate for a respectiveapplication event comprises: determining daily counts of instances ofthe respective application event for which data is received; calculatinga statistic based on the daily counts; and using the statistic to selectthe sampling rate.
 17. The method of claim 16, wherein the statistic isa median value of the daily counts.
 18. The method of claim 1, whereinproviding the report comprises generating a dashboard showinginformation characterizing the one or more sets.
 19. A server system,comprising: one or more processors; and memory storing one or moreprograms for execution by the one or more processors, the one or moreprograms including instructions for: receiving usage data for aplurality of applications on client devices, wherein the usage dataincludes data for instances of each of a plurality of application eventsthat occur in the plurality of applications, in response to usersinteracting with the plurality of applications on the client devices;storing the usage data in a first data store; determining, forrespective applications of the plurality of applications, sampling ratesfor respective application events of the plurality of application eventsbased at least in part on respective quantities of instances of therespective application events during a period of time; after determiningthe sampling rates: sampling, from the first data store, the data forthe instances of the respective application events at the sampling ratesto form respective sets of usage data, wherein each set of usage dataincludes data samples corresponding to distinct instances of arespective application event; storing the sets of usage data in a seconddata store; and providing a report using one or more sets of the sets ofusage data.
 20. A non-transitory computer-readable storage medium,storing one or more programs configured for execution by one or moreprocessors of a server system, the one or more programs includinginstructions, which when executed by the one or more processors causethe server system to: receive usage data for a plurality of applicationson client devices, wherein the usage data includes data for instances ofeach of a plurality of application events that occur in the plurality ofapplications, in response to users interacting with the plurality ofapplications on the client devices; store the usage data in a first datastore; determine, for respective applications of the plurality ofapplications, sampling rates for respective application events of theplurality of application events based at least in part on respectivequantities of instances of the respective application events during aperiod of time; after determining the sampling rates: sample, from thefirst data store, the data for the instances of the respectiveapplication events at the sampling rates to form respective sets ofusage data, wherein each set of usage data includes data samplescorresponding to distinct instances of a respective application event;store the sets of usage data in a second data store; and provide areport using one or more sets of the sets of usage data.